Systems and methods for detecting traffic lights of driving lanes using a camera and multiple models

ABSTRACT

System, methods, and other embodiments described herein relate to improving the detection of traffic lights associated with a driving lane using a camera instead of map data. In one embodiment, a method includes estimating, from an image using a first model, depth and orientation information of traffic lights relative to a driving lane of a vehicle. The method also includes computing, using a second model, relevancy scores for the traffic lights according to geometric inferences between the depth and the orientation information. The method also includes assigning, using the second model, a primary relevancy score for a light of the traffic lights associated with the driving lane according to the depth and the orientation information. The method also includes executing a control task by the vehicle according to the primary relevancy score and a state confidence, computed by the first model, for the light.

TECHNICAL FIELD

The subject matter described herein relates, in general, to detectingtraffic lights, and, more particularly, to detecting traffic lightsassociated with a driving lane using a camera instead of map data.

BACKGROUND

Vehicle systems use data from positioning and other sensors forperforming guidance and other tasks. Guidance operations may involvevarious tasks involving a navigation system coordinating with othersystems that assist a vehicle, such as an automated driving system(ADS). For example, vehicle systems use map and sensor data to guide avehicle through an intersection. As part of the guidance, a machinelearning (ML) model for the ADS may help assist with driving tasksthrough the intersection and similar risky scenarios by perceivingobjects. Assistance from the ADS can help improve safety and comfort foran operator.

In various implementations, the vehicle systems use high-definition (HD)map data when assisting navigation through intersections. HD map datacan increase the accuracy of tasks, thereby improving safety. However,ADS and other systems relying on HD map data increase computationalcosts from added complexity. Furthermore, HD map data may be unavailableand stale in certain geographic areas. This can create difficulties whenassistance from the vehicle systems is unavailable, particularly forintersections.

SUMMARY

In one embodiment, example systems and methods relate to a manner ofdetecting traffic lights associated with a driving lane using a camerainstead of map data. In various implementations, systems detectingtraffic lights to navigate through an intersection utilize map andsensor data. The reliance on map data, particularly high-definition (HD)map data, can increase computational costs for navigation and othervehicle systems from increased complexity. The unavailability of mapdata in certain regions also impacts safety for vehicle systems relyingon the data for providing assistance. Therefore, in one embodiment, adetection system processes an image from a camera (e.g., monocularcamera) in a first model (e.g., a neural network, an encoder-decoderarchitecture, etc.) for estimating characteristics (e.g., depth,location, color, etc.) of traffic lights that may be associated with adriving lane. In particular, the first model may estimate depth andorientation information for traffic lights (e.g., multi-lamp signals,railroad signals, crosswalk signals, etc.) within the image instead ofrelying on map data. A second model may aggregate the depth andorientation information to accurately compute relevancy scores for thetraffic lights using geometric processing. Here, the relevancy scoresmay indicate which traffic lights in the image are most relevant to thevehicle for given driving lanes. In this way, vehicle systems canreliably execute tasks involving the traffic lights along the drivinglane using the relevancy scores. Accordingly, the detection systemprocesses an image and uses multiple models (e.g., a two-stage model)for accurately scoring relevant traffic lights instead of using mapdata, thereby reducing complexity and increasing assistanceavailability.

In one embodiment, a detection system for detecting traffic lightsassociated with a driving lane using a camera instead of map data isdisclosed. The detection system includes a processor and memory storinginstructions that, when executed by the processor, cause the processorto estimate, from an image using a first model, depth and orientationinformation of the traffic lights relative to a driving lane of avehicle. The instructions also include instructions to compute, using asecond model, relevancy scores for the traffic lights according togeometric inferences between the depth and the orientation information.The instructions also include instructions to assign, using the secondmodel, a primary relevancy score for a light of the traffic lightsassociated with the driving lane according to the depth and theorientation information. The instructions also include instructions toexecute a control task by the vehicle according to the primary relevancyscore and a state confidence, computed by the first model, for thelight.

In one embodiment, a non-transitory computer-readable medium fordetecting traffic lights associated with a driving lane using a camerainstead of map data and including instructions that when executed by aprocessor cause the processor to perform one or more functions isdisclosed. The instructions include instructions to estimate, from animage using a first model, depth and orientation information of trafficlights relative to a driving lane of a vehicle. The instructions alsoinclude instructions to compute, using a second model, relevancy scoresfor the traffic lights according to geometric inferences between thedepth and the orientation information. The instructions also includeinstructions to assign, using the second model, a primary relevancyscore for a light of the traffic lights associated with the driving laneaccording to the depth and the orientation information. The instructionsalso include instructions to execute a control task by the vehicleaccording to the primary relevance score and a state confidence,computed by the first model, for the light.

In one embodiment, a method for detecting traffic lights associated witha driving lane using a camera instead of map data is disclosed. In oneembodiment, the method includes estimating, from an image using a firstmodel, depth and orientation information of traffic lights relative to adriving lane of a vehicle. The method also includes computing, using asecond model, relevancy scores for the traffic lights according togeometric inferences between the depth and the orientation information.The method also includes assigning, using the second model, a primaryrelevancy score for a light of the traffic lights associated with thedriving lane according to the depth and the orientation information. Themethod also includes executing a control task by the vehicle accordingto the primary relevancy score and a state confidence, computed by thefirst model, for the light.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate various systems, methods, andother embodiments of the disclosure. It will be appreciated that theillustrated element boundaries (e.g., boxes, groups of boxes, or othershapes) in the figures represent one embodiment of the boundaries. Insome embodiments, one element may be designed as multiple elements ormultiple elements may be designed as one element. In some embodiments,an element shown as an internal component of another element may beimplemented as an external component and vice versa. Furthermore,elements may not be drawn to scale.

FIG. 1 illustrates one embodiment of a vehicle within which systems andmethods disclosed herein may be implemented.

FIG. 2 illustrates one embodiment of a detection system that isassociated with identifying traffic lights associated with a drivinglane using a camera instead of map data.

FIG. 3 illustrates one embodiment of the detection system of FIG. 2using an encoder and decoders that detect through probabilisticformulations traffic lights associated with the driving lane.

FIG. 4 illustrates one embodiment of a method that is associated withidentifying traffic lights for the driving lane by computing relevancyscores.

FIG. 5 illustrates an example of detected orientations for trafficlights relative to the driving lane.

DETAILED DESCRIPTION

Systems, methods, and other embodiments associated with improving thedetection of traffic lights associated with a driving lane using acamera instead of map data are disclosed herein. In variousimplementations, detecting traffic lights to navigate through anintersection using map and sensor data can encounter difficulties. Forinstance, systems that use high-definition (HD) map data for associatinga traffic light, regardless of state, to a driving lane encountersignificant computational costs from complex structures that form HD mapdata. Furthermore, map data may be available in certain regions. Vehiclesystems operating in regions where map data is unavailable or staleimpacts safety for driver assistance systems that rely on map data.Therefore, in one embodiment, a detection system uses a first model(e.g., neural network (NN), perceptron, convolutional NN (CNN), etc.) toestimate depth and orientation information of traffic lights from animage (e.g., a monocular image) taken by a vehicle-based camera. Here,the depth and orientation information give the detection system insightinto the relative position and geometries of the traffic lights (e.g.,multi-lamp signals, railroad signals, crosswalk signals, etc.) for adriving lane substantially independent of map data. For example, thegeometries can indicate an angle between the traffic lights and thevehicle relative to the driving lane. Regarding further processing, thedetection system uses a second model (e.g., perceptron, CNN, etc.) thatcomputes relevancy scores for the traffic lights in association with thedriving lane. This may involve the second model forming geometricinferences between the depth and the orientation information.Furthermore, the detection system uses the second model to predict andassign a primary relevance score for a traffic light most likelyassociated with the driving lane using the depth and the orientationinformation.

Moreover, a vehicle system may execute a control using the primaryrelevance score and a state confidence, computed by the first model, forthe relevant traffic light. Here, the state confidence indicatesprobabilities of a color and a shape associated with the traffic light.In one approach, the detection system assigns the primary relevancescore using limited map data associated with the driving lane or thedriving scenario (e.g., intersection crossing, vehicle-following, etc.).Accordingly, a detection system uses multi-stage modeling to derivegeometric relations between traffic lights and a driving lane andcompute a relevancy score using predictions independent of map data,thereby improving accessibility and robustness.

Referring to FIG. 1 , an example of a vehicle 100 is illustrated. Asused herein, a “vehicle” is any form of motorized transport. In one ormore implementations, the vehicle 100 is an automobile. Whilearrangements will be described herein with respect to automobiles, itwill be understood that embodiments are not limited to automobiles. Insome implementations, a detection system uses road-side units (RSU),consumer electronics (CE), mobile devices, robots, drones, and so onthat benefit from the functionality discussed herein associated withimproving the detection of traffic lights corresponding to a drivinglane using a camera instead of map data.

The vehicle 100 also includes various elements. It will be understoodthat in various embodiments, the vehicle 100 may have less than theelements shown in FIG. 1 . The vehicle 100 can have any combination ofthe various elements shown in FIG. 1 . Furthermore, the vehicle 100 canhave additional elements to those shown in FIG. 1 . In somearrangements, the vehicle 100 may be implemented without one or more ofthe elements shown in FIG. 1 . While the various elements are shown asbeing located within the vehicle 100 in FIG. 1 , it will be understoodthat one or more of these elements can be located external to thevehicle 100. Furthermore, the elements shown may be physically separatedby large distances. For example, as discussed, one or more components ofthe disclosed system can be implemented within a vehicle while furthercomponents of the system are implemented within a cloud-computingenvironment or other system that is remote from the vehicle 100.

Some of the possible elements of the vehicle 100 are shown in FIG. 1 andwill be described along with subsequent figures. However, a descriptionof many of the elements in FIG. 1 will be provided after the discussionof FIGS. 2-5 for purposes of brevity of this description. Additionally,it will be appreciated that for simplicity and clarity of illustration,where appropriate, reference numerals have been repeated among thedifferent figures to indicate corresponding or analogous elements. Inaddition, the discussion outlines numerous specific details to provide athorough understanding of the embodiments described herein. Those ofskill in the art, however, will understand that the embodimentsdescribed herein may be practiced using various combinations of theseelements. In either case, the vehicle 100 includes a detection system170 that is implemented to perform methods and other functions asdisclosed herein relating to improving the detection of traffic lightsassociated with a driving lane using a camera instead of map data. Aswill be discussed in greater detail subsequently, the detection system170, in various embodiments, is implemented partially within the vehicle100, and as a cloud-based service. For example, in one approach,functionality associated with at least one module of the detectionsystem 170 is implemented within the vehicle 100 while furtherfunctionality is implemented within a cloud-based computing system.

With reference to FIG. 2 , one embodiment of the detection system 170 ofFIG. 1 is further illustrated. The detection system 170 is shown asincluding a processor(s) 110 from the vehicle 100 of FIG. 1 .Accordingly, the processor(s) 110 may be a part of the detection system170, the detection system 170 may include a separate processor from theprocessor(s) 110 of the vehicle 100, or the detection system 170 mayaccess the processor(s) 110 through a data bus or another communicationpath. In one embodiment, the detection system 170 includes a memory 210that stores an estimation module 220. The memory 210 is a random-accessmemory (RAM), a read-only memory (ROM), a hard-disk drive, a flashmemory, or other suitable memory for storing the estimation module 220.The estimation module 220 is, for example, computer-readableinstructions that when executed by the processor(s) 110 cause theprocessor(s) 110 to perform the various functions disclosed herein.

The detection system 170 as illustrated in FIG. 2 is generally anabstracted form of the detection system 170. Furthermore, the estimationmodule 220 generally includes instructions that function to control theprocessor(s) 110 to receive data inputs from one or more sensors of thevehicle 100. The inputs are, in one embodiment, observations of one ormore objects in an environment proximate to the vehicle 100 and/or otheraspects about the surroundings. As provided for herein, the estimationmodule 220, in one embodiment, acquires the sensor data 250 thatincludes at least camera images (e.g., monocular images). In furtherarrangements, the estimation module 220 acquires the sensor data 250from further sensors such as radar sensors 123, LIDAR sensors 124, andother sensors as may be suitable for identifying and locating objects(e.g., vehicles).

Accordingly, the estimation module 220, in one embodiment, controls therespective sensors to provide the data inputs in the form of the sensordata 250. Additionally, while the estimation module 220 is discussed ascontrolling the various sensors to provide the sensor data 250, in oneor more embodiments, the estimation module 220 can employ othertechniques to acquire the sensor data 250 that are either active orpassive. Moreover, the estimation module 220 can undertake variousapproaches to fuse data from multiple sensors when providing the sensordata 250 and/or from sensor data acquired over a wireless communicationlink. Thus, the sensor data 250, in one embodiment, represents acombination of perceptions acquired from multiple sensors.

Moreover, in one embodiment, the detection system 170 includes a datastore 230. In one embodiment, the data store 230 is a database. Thedatabase is, in one embodiment, an electronic data structure stored inthe memory 210 or another data store and that is configured withroutines that can be executed by the processor(s) 110 for analyzingstored data, providing stored data, organizing stored data, and so on.Thus, in one embodiment, the data store 230 stores data used by theestimation module in executing various functions. In one embodiment, thedata store 230 includes the sensor data 250 along with, for example,metadata that characterize various aspects of the sensor data 250. Forexample, the metadata can include location coordinates (e.g., longitudeand latitude), relative map coordinates or tile identifiers, time/datestamps from when the separate sensor data 250 was generated, and so on.In one embodiment, the data store 230 further includes relevancy scores240. These scores may be an index (e.g., 1-10), a percentage, a scale(0-1), and so on that the detection system 170 computes. The relevancyscore can indicate the importance of an upcoming traffic light (e.g.,multi-lamp signals, railroad signals, crosswalk signals, etc.) detectedwithin an image to the driving lane of a vehicle. In one approach, thetraffic light is scored entirely rather than by bulb. For instance, anentire traffic light may include the bulb housing, support pole, supportcable, and so on associated with the traffic light structure.Furthermore, the relevancy score may be aggregated so that the scorereflects a number (e.g., 2, 3, etc.) of traffic lights relevant to thedriving lane.

In various implementations, the estimation module 220 is furtherconfigured to perform additional tasks beyond controlling the respectivesensors to acquire and provide the sensor data 250. For example, theestimation module 220 includes instructions that cause the processor(s)110 to estimate from an image, such as a monocular image taken by one ofthe cameras 126, depth and orientation information of traffic lightsusing a first model (e.g., NN, perceptron, CNN, etc.). Such estimatesare relative to a driving lane of the vehicle 100 (e.g., an ego vehicle)so that the detection system 170 identifies the most relevant trafficlights. In various implementations, the orientation information is afacing degree of a camera or the vehicle 100 relative to one or moretraffic lights. For instance, as explained below, an orientation of 0 or90 degrees reflects that one or more traffic lights are directlyassociated with the current driving lane for the vehicle 100. On thecontrary, an orientation of 45 or 135 degrees may reflect that one ormore traffic lights are associated with other lanes and less relevant tothe vehicle 100.

In one approach, the estimates also include sizing information havingcoordinates that indicate x-y position, height (h), and width (w)geometries of the traffic lights within the image, such as relative toimage pixels. For example, the h and w are dimensions of a bounding boxencompassing the traffic lights or other objects. As explained below,the depth, the orientation information, and the size information may beused by the detection system 170 to determine the most relevant trafficlights for the vehicle 100.

Moreover, a second model (e.g., perceptron, CNN, etc.) can computerelevancy scores for one or more traffic lights associated with adriving lane from geometric inferences between the depth, theorientation, and the size information. For instance, a potential trafficlight 0 degrees and 20×5 pixels at a depth ahead of an ado vehicle tothe vehicle 100 is more relevant than one 45 degrees and 30×5 pixels. Inone approach, the detection system 170 factors motion estimates fromother vehicle systems when computing relevancy scores. For example, thetraffic light at 45 degrees is more relevant than a different trafficlight at 0 degrees when a lane-change maneuver is likely by the ADS forthe vehicle 100.

In addition, the detection system 170 may assign a primary relevancyscore for a light of the traffic lights associated with the driving laneusing the second model with limited assistance from map data. In oneapproach, the primary relevancy score is inferred from the depth and theorientation information. In this way, the detection system 170associates a primary light for a driving lane and outputs a relatedconfidence score independent of map data, thereby reducing computationcosts. The output may also include a location and a color for theprimary light associated with the confidence score specific to thedriving lane. For additional insight, the detection system 170 detects asecondary relevancy score for another light in an adjacent lane to thevehicle 100 for advanced tasks, such as a lane crossing by an ADS.

Now turning to FIG. 3 , one embodiment of the detection system 170 ofFIG. 2 using an encoder and decoders 300 that detect throughprobabilistic formulations traffic lights associated with a driving laneis illustrated. Here, the detection system 170 identifies traffic lightsusing images from a camera substantially independent of map data. Inparticular, the first model 310 (e.g., NN, perceptron, CNN, etc.)includes an encoder and decoders to unify the detection of trafficlights, such as by decoding data from a common encoder to predict depth,orientation, size, and recognition in parallel within the first model.In this way, the detection system 170 improves efficiency and reducescomputational costs.

Moreover, an encoder 320 may be a vision backbone that detects objectfeatures or attributes within an image taken from a single view, such asby one monocular camera. The depth head 330 may decode the features andoutput the depth map 340 using features from the encoder 320. Forexample, the depth map 340 is a heat map representing distance and canbe color-coded according to object type (e.g., vehicle, pedestrian,illuminated device, etc.).

Regarding the orientation head 350, this decoder may process the outputof the encoder 320 to estimate a degree (e.g., facing degree) betweenthe vehicle 100 and objects (e.g., traffic lights) relative to a drivinglane. As previously explained, a facing degree can represent a geometricmeasurement between a camera or the vehicle 100 in a driving lanerelative to one or more traffic lights. For instance, an orientation of0 or 90 degrees reflects that one or more traffic lights are directlyassociated with the current driving lane for the vehicle 100. On thecontrary, an orientation of 135 or 45 degrees may reflect that one ormore traffic lights are associated with other lanes and less relevant tothe vehicle 100.

In one approach, the first model 310 includes the detection head 360that locates a traffic light and computes coordinates indicating x-yposition, h, and w geometries within an image, such as relative topixels. For example, the h and w geometries are dimensions of a boundingbox at position x-y encompassing a traffic light unit. Furthermore,these locations may be for an entire traffic light instead of individualbulbs for reducing computational loads to the detection system 170and/or the estimation module 220. For instance, an entire traffic lightincludes the bulb housing, support pole, support cable, and so onassociated with the traffic light structure.

Regarding additional detection tasks, the first model 310 includes therecognition head 380 that determines states of potential traffic lights.For example, the recognition head 380 decodes the output of encoder 320and outputs red, green, yellow, symbol (e.g., arrow), and so oninformation. An associated confidence score for potential traffic lightswithin the image can also form part of the information. Thus, thedetection system 170 outputs a state confidence indicating probabilitiesof a color and a shape for a light cover, bulb, housing, and so onassociated with traffic lights.

In various implementations, the aggregator 370 is a second model thatprocesses outputs from the depth head 330, the orientation head 350, andthe detection head 360. For instance, the second model is a subnetwork(e.g., neural network, heuristic method, etc.) that outputs relevantscores for one or more traffic lights associated with a driving lane(e.g., ego lane). A score can indicate probabilities or measuresrelating a traffic light to a driving lane. For example, an image (e.g.,monocular image) has two traffic lights, where the first traffic lighthas a 0.9 score and the second traffic light 0.1 score from therelevancy scores 240. As such, the first traffic light may be mostrelevant for the vehicle 100. Here, the aggregator 370 reliably scorestraffic lights by understanding the geometries of the traffic sceneassociated with the driving lane from the depth, orientation, anddetection information.

Furthermore, the detection system 170 may communicate the scoresoutputted from the aggregator 370 and information from the recognitionhead 380 to downstream processes that perform tasks such as objectdetection, object tracking, and motion planning. For example, aperception system tracking a vehicle perpendicular to the vehicle 100near an intersection infers a speed reduction using the detection system170. Accordingly, the detection system 170 may compute that the primaryrelevancy score is 0.8, state green, and state confidence 0.9 for atraffic light. In other words, the vehicle approaching the intersectionwill reduce speed and stop since the vehicle 100 is likely travelingtowards the intersection with a green light, thereby reducing collisionrisk.

Turning now to FIG. 4 , a flowchart of a method 400 that is associatedwith improving the detection of traffic lights associated with a drivinglane using a camera instead of map data is illustrated. Method 400 willbe discussed from the perspective of the detection system 170 of FIGS. 1and 2 . While method 400 is discussed in combination with the detectionsystem 170, it should be appreciated that the method 400 is not limitedto being implemented within the detection system 170 but is instead oneexample of a system that may implement the method 400.

At 410, the detection system 170 estimates depth and orientation oftraffic lights (e.g., multi-lamp signals, railroad signals, crosswalksignals, etc.) relative to a driving lane for the vehicle 100 using afirst model (e.g., NN, perceptron, CNN, etc.). As previously explained,the detection system 170 estimates from an image (e.g., a monocularimage) depth and orientation information of traffic lights relative to adriving lane of the vehicle 100 (e.g., an ego vehicle) using the firstmodel for relevance. Here, depth information can be a depth map ofvarious objects (e.g., vehicle, pedestrian, illuminated device, etc.).Furthermore, the orientation information may be a facing degree of acamera or the vehicle 100 relative to one or more traffic lights. Forinstance, an orientation of 0 or 90 degrees reflects that one or moretraffic lights are directly associated with the current driving lane forthe vehicle 100. On the contrary, an orientation of 45 or 135 degreesmay reflect traffic lights associated with other lanes that are lessrelevant to the vehicle 100.

Regarding relevancy, at 420 the detection system 170 computes relevancyscores for traffic lights from the depth and the orientation informationusing a second model (e.g., perceptron, CNN, etc.). Here, the relevancyscores can be computed for traffic lights associated with a driving lanefrom geometric predictions and inferences between the depth and theorientation information. For instance, a traffic light at 0 degrees tothe vehicle 100 is more relevant than at 45 degrees. In one approach,the detection system 170 factors motion estimates from other vehiclesystems when computing relevancy scores. For example, the traffic lightat 45 degrees is more relevant than a different traffic light at 0degrees when a lane change is planned by the ADS for the vehicle 100.

At 430, the detection system 170 assigns a primary relevancy score for atraffic light according to depth and orientation using the second model.In particular, the detection system 170 associates the traffic lightwith the driving lane with limited assistance from map data. In oneapproach, the primary relevancy score is associated with the depth andthe orientation information as part of geometric computations. Foradditional insight, the detection system 170 detects a secondaryrelevancy score for another light in an adjacent lane to the vehicle 100for advanced tasks, such as a lane crossing. In this way, the detectionsystem 170 associates a primary light for a driving lane and outputsrelated confidence scores independent of map data, thereby reducingcomputation costs.

At 440, a system of the vehicle 100 executes a control task according tothe primary relevancy score and a state confidence. Here, the firstmodel determines states of potential traffic lights and outputs red,green, yellow, symbol (e.g., arrow), and so on information including anassociated confidence score for potential traffic lights within theimage. As such, the detection system 170 outputs a state confidenceindicating probabilities of a color and a shape of a light cover, bulb,housing, and so on associated with the traffic lights for other tasks.

Moreover, a control task may involve object detection, object tracking,and motion planning. For example, a perception system monitoring across-track scenario near an intersection for the vehicle 100 infersusing the detection system 170 that another vehicle will reduce speed.Here, the detection system 170 can compute that the primary relevancyscore is 0.8, state green, and state confidence 0.9 for a traffic light.In other words, the crossing vehicle may reduce speed and stop since thevehicle 100 is likely traveling towards an intersection with a greenlight. Accordingly, the detection system 170 improves efficiency andreduces computational costs through the first model unifying detectionof traffic lights and the second model computing relevancy.

Now turning to FIG. 5 , an example 500 of detected orientations fortraffic lights relative to the driving lane is illustrated. Here, thevehicle 100 is approaching the traffic lights 510 ₁ and 510 ₂ from thedriving lane 520. The detection system 170 may use the first model toestimate from an image that the traffic light 510 ₁ is 90 degreesrelative to a unit circle for the vehicle 100 and green. Similarly, theorientation computation for the traffic light 510 ₂ is 135 degrees. Assuch, the traffic light 510 ₁ may be scored as more relevant than thetraffic light 510 ₂ by the second model due to orientation or facingdegrees. In one approach, the detection system 170 detects the trafficlight 510 ₂ at 0 degrees and 510 ₁ at 45 degrees when the vehicle 100approaches from the driving lane 530. Accordingly, the traffic light 510₂ may be scored as more relevant than the traffic light 510 ₁ in thisdriving scenario when traveling on the driving lane 530.

FIG. 1 will now be discussed in full detail as an example environmentwithin which the system and methods disclosed herein may operate. Insome instances, the vehicle 100 is configured to switch selectivelybetween different modes of operation/control according to the directionof one or more modules/systems of the vehicle 100. In one approach, themodes include: 0, no automation; 1, driver assistance; 2, partialautomation; 3, conditional automation; 4, high automation; and 5, fullautomation. In one or more arrangements, the vehicle 100 can beconfigured to operate in a subset of possible modes.

In one or more embodiments, the vehicle 100 is an automated orautonomous vehicle. As used herein, “autonomous vehicle” refers to avehicle that is capable of operating in an autonomous mode (e.g.,category 5, full automation). “Automated mode” or “autonomous mode”refers to navigating and/or maneuvering the vehicle 100 along a travelroute using one or more computing systems to control the vehicle 100with minimal or no input from a human driver. In one or moreembodiments, the vehicle 100 is highly automated or completelyautomated. In one embodiment, the vehicle 100 is configured with one ormore semi-autonomous operational modes in which one or more computingsystems perform a portion of the navigation and/or maneuvering of thevehicle along a travel route, and a vehicle operator (i.e., driver)provides inputs to the vehicle to perform a portion of the navigationand/or maneuvering of the vehicle 100 along a travel route.

The vehicle 100 can include one or more processors 110. In one or morearrangements, the processor(s) 110 can be a main processor of thevehicle 100. For instance, the processor(s) 110 can be an electroniccontrol unit (ECU), an application-specific integrated circuit (ASIC), amicroprocessor, etc. The vehicle 100 can include one or more data stores115 for storing one or more types of data. The data store(s) 115 caninclude volatile and/or non-volatile memory. Examples of suitable datastores 115 include RAM, flash memory, ROM, Programmable Read-Only Memory(PROM), Erasable Programmable Read-Only Memory (EPROM), ElectricallyErasable Programmable Read-Only Memory (EEPROM), registers, magneticdisks, optical disks, and hard drives. The data store(s) 115 can be acomponent of the processor(s) 110, or the data store(s) 115 can beoperatively connected to the processor(s) 110 for use thereby. The term“operatively connected,” as used throughout this description, caninclude direct or indirect connections, including connections withoutdirect physical contact.

In one or more arrangements, the one or more data stores 115 can includemap data 116. The map data 116 can include maps of one or moregeographic areas. In some instances, the map data 116 can includeinformation or data on roads, traffic control devices, road markings,structures, features, and/or landmarks in the one or more geographicareas. The map data 116 can be in any suitable form. In some instances,the map data 116 can include aerial views of an area. In some instances,the map data 116 can include ground views of an area, including360-degree ground views. The map data 116 can include measurements,dimensions, distances, and/or information for one or more items includedin the map data 116 and/or relative to other items included in the mapdata 116. The map data 116 can include a digital map with informationabout road geometry.

In one or more arrangements, the map data 116 can include one or moreterrain maps 117. The terrain map(s) 117 can include information aboutthe terrain, roads, surfaces, and/or other features of one or moregeographic areas. The terrain map(s) 117 can include elevation data inthe one or more geographic areas. The terrain map(s) 117 can define oneor more ground surfaces, which can include paved roads, unpaved roads,land, and other things that define a ground surface.

In one or more arrangements, the map data 116 can include one or morestatic obstacle maps 118. The static obstacle map(s) 118 can includeinformation about one or more static obstacles located within one ormore geographic areas. A “static obstacle” is a physical object whoseposition does not change or substantially change over a period of timeand/or whose size does not change or substantially change over a periodof time. Examples of static obstacles can include trees, buildings,curbs, fences, railings, medians, utility poles, statues, monuments,signs, benches, furniture, mailboxes, large rocks, or hills. The staticobstacles can be objects that extend above ground level. The one or morestatic obstacles included in the static obstacle map(s) 118 can havelocation data, size data, dimension data, material data, and/or otherdata associated with it. The static obstacle map(s) 118 can includemeasurements, dimensions, distances, and/or information for one or morestatic obstacles. The static obstacle map(s) 118 can be high qualityand/or highly detailed. The static obstacle map(s) 118 can be updated toreflect changes within a mapped area.

One or more data stores 115 can include sensor data 119. In thiscontext, “sensor data” means any information about the sensors that thevehicle 100 is equipped with, including the capabilities and otherinformation about such sensors. As will be explained below, the vehicle100 can include the sensor system 120. The sensor data 119 can relate toone or more sensors of the sensor system 120. As an example, in one ormore arrangements, the sensor data 119 can include information about oneor more LIDAR sensors 124 of the sensor system 120.

In some instances, at least a portion of the map data 116 and/or thesensor data 119 can be located in one or more data stores 115 locatedonboard the vehicle 100. Alternatively, or in addition, at least aportion of the map data 116 and/or the sensor data 119 can be located inone or more data stores 115 that are located remotely from the vehicle100.

As noted above, the vehicle 100 can include the sensor system 120. Thesensor system 120 can include one or more sensors. “Sensor” means adevice that can detect, and/or sense something. In at least oneembodiment, the one or more sensors detect, and/or sense in real-time.As used herein, the term “real-time” means a level of processingresponsiveness that a user or system senses as sufficiently immediatefor a particular process or determination to be made, or that enablesthe processor to keep up with some external process.

In arrangements in which the sensor system 120 includes a plurality ofsensors, the sensors may function independently or two or more of thesensors may function in combination. The sensor system 120 and/or theone or more sensors can be operatively connected to the processor(s)110, the data store(s) 115, and/or another element of the vehicle 100.The sensor system 120 can produce observations about a portion of theenvironment of the vehicle 100 (e.g., nearby vehicles).

The sensor system 120 can include any suitable type of sensor. Variousexamples of different types of sensors will be described herein.However, it will be understood that the embodiments are not limited tothe particular sensors described. The sensor system 120 can include oneor more vehicle sensors 121. The vehicle sensor(s) 121 can detectinformation about the vehicle 100 itself. In one or more arrangements,the vehicle sensor(s) 121 can be configured to detect position andorientation changes of the vehicle 100, such as, for example, based oninertial acceleration. In one or more arrangements, the vehiclesensor(s) 121 can include one or more accelerometers, one or moregyroscopes, an inertial measurement unit (IMU), a dead-reckoning system,a global navigation satellite system (GNSS), a global positioning system(GPS), a navigation system 147, and/or other suitable sensors. Thevehicle sensor(s) 121 can be configured to detect one or morecharacteristics of the vehicle 100 and/or a manner in which the vehicle100 is operating. In one or more arrangements, the vehicle sensor(s) 121can include a speedometer to determine a current speed of the vehicle100.

Alternatively, or in addition, the sensor system 120 can include one ormore environment sensors 122 configured to acquire data about anenvironment surrounding the vehicle 100 in which the vehicle 100 isoperating. “Surrounding environment data” includes data about theexternal environment in which the vehicle is located or one or moreportions thereof. For example, the one or more environment sensors 122can be configured to sense obstacles in at least a portion of theexternal environment of the vehicle 100 and/or data about suchobstacles. Such obstacles may be stationary objects and/or dynamicobjects. The one or more environment sensors 122 can be configured todetect other things in the external environment of the vehicle 100, suchas, for example, lane markers, signs, traffic lights, traffic signs,lane lines, crosswalks, curbs proximate the vehicle 100, off-roadobjects, etc.

Various examples of sensors of the sensor system 120 will be describedherein. The example sensors may be part of the one or more environmentsensors 122 and/or the one or more vehicle sensors 121. However, it willbe understood that the embodiments are not limited to the particularsensors described.

As an example, in one or more arrangements, the sensor system 120 caninclude one or more of: radar sensors 123, LIDAR sensors 124, sonarsensors 125, weather sensors, haptic sensors, locational sensors, and/orone or more cameras 126. In one or more arrangements, the one or morecameras 126 can be high dynamic range (HDR) cameras, stereo, or infrared(IR) cameras.

The vehicle 100 can include an input system 130. An “input system”includes components or arrangement or groups thereof that enable variousentities to enter data into a machine. The input system 130 can receivean input from a vehicle occupant. The vehicle 100 can include an outputsystem 135. An “output system” includes one or more components thatfacilitate presenting data to a vehicle occupant.

The vehicle 100 can include one or more vehicle systems 140. Variousexamples of the one or more vehicle systems 140 are shown in FIG. 1 .However, the vehicle 100 can include more, fewer, or different vehiclesystems. It should be appreciated that although particular vehiclesystems are separately defined, any of the systems or portions thereofmay be otherwise combined or segregated via hardware and/or softwarewithin the vehicle 100. The vehicle 100 can include a propulsion system141, a braking system 142, a steering system 143, a throttle system 144,a transmission system 145, a signaling system 146, and/or a navigationsystem 147. Any of these systems can include one or more devices,components, and/or a combination thereof, now known or later developed.

The navigation system 147 can include one or more devices, applications,and/or combinations thereof, now known or later developed, configured todetermine the geographic location of the vehicle 100 and/or to determinea travel route for the vehicle 100. The navigation system 147 caninclude one or more mapping applications to determine a travel route forthe vehicle 100. The navigation system 147 can include a globalpositioning system, a local positioning system, or a geolocation system.

The processor(s) 110, the detection system 170, and/or the automateddriving module(s) 160 can be operatively connected to communicate withthe various vehicle systems 140 and/or individual components thereof.For example, returning to FIG. 1 , the processor(s) 110 and/or theautomated driving module(s) 160 can be in communication to send and/orreceive information from the various vehicle systems 140 to control themovement of the vehicle 100. The processor(s) 110, the detection system170, and/or the automated driving module(s) 160 may control some or allof the vehicle systems 140 and, thus, may be partially or fullyautonomous as defined by the society of automotive engineers (SAE)levels 0 to 5.

The processor(s) 110, the detection system 170, and/or the automateddriving module(s) 160 can be operatively connected to communicate withthe various vehicle systems 140 and/or individual components thereof.For example, returning to FIG. 1 , the processor(s) 110, the detectionsystem 170, and/or the automated driving module(s) 160 can be incommunication to send and/or receive information from the variousvehicle systems 140 to control the movement of the vehicle 100. Theprocessor(s) 110, the detection system 170, and/or the automated drivingmodule(s) 160 may control some or all of the vehicle systems 140.

The processor(s) 110, the detection system 170, and/or the automateddriving module(s) 160 may be operable to control the navigation andmaneuvering of the vehicle 100 by controlling one or more of the vehiclesystems 140 and/or components thereof. For instance, when operating inan autonomous mode, the processor(s) 110, the detection system 170,and/or the automated driving module(s) 160 can control the directionand/or speed of the vehicle 100. The processor(s) 110, the detectionsystem 170, and/or the automated driving module(s) 160 can cause thevehicle 100 to accelerate, decelerate, and/or change direction. As usedherein, “cause” or “causing” means to make, force, compel, direct,command, instruct, and/or enable an event or action to occur or at leastbe in a state where such event or action may occur, either in a director indirect manner.

The vehicle 100 can include one or more actuators 150. The actuators 150can be an element or a combination of elements operable to alter one ormore of the vehicle systems 140 or components thereof responsive toreceiving signals or other inputs from the processor(s) 110 and/or theautomated driving module(s) 160. For instance, the one or more actuators150 can include motors, pneumatic actuators, hydraulic pistons, relays,solenoids, and/or piezoelectric actuators, just to name a fewpossibilities.

The vehicle 100 can include one or more modules, at least some of whichare described herein. The modules can be implemented ascomputer-readable program code that, when executed by a processor(s)110, implement one or more of the various processes described herein.One or more of the modules can be a component of the processor(s) 110,or one or more of the modules can be executed on and/or distributedamong other processing systems to which the processor(s) 110 isoperatively connected. The modules can include instructions (e.g.,program logic) executable by one or more processors 110. Alternatively,or in addition, one or more data stores 115 may contain suchinstructions.

In one or more arrangements, one or more of the modules described hereincan include artificial intelligence elements, e.g., neural network,fuzzy logic, or other machine learning algorithms. Furthermore, in oneor more arrangements, one or more of the modules can be distributedamong a plurality of the modules described herein. In one or morearrangements, two or more of the modules described herein can becombined into a single module.

The vehicle 100 can include one or more automated driving modules 160.The automated driving module(s) 160 can be configured to receive datafrom the sensor system 120 and/or any other type of system capable ofcapturing information relating to the vehicle 100 and/or the externalenvironment of the vehicle 100. In one or more arrangements, theautomated driving module(s) 160 can use such data to generate one ormore driving scene models. The automated driving module(s) 160 candetermine position and velocity of the vehicle 100. The automateddriving module(s) 160 can determine the location of obstacles,obstacles, or other environmental features including traffic signs,trees, shrubs, neighboring vehicles, pedestrians, etc.

The automated driving module(s) 160 can be configured to receive, and/ordetermine location information for obstacles within the externalenvironment of the vehicle 100 for use by the processor(s) 110, and/orone or more of the modules described herein to estimate position andorientation of the vehicle 100, vehicle position in global coordinatesbased on signals from a plurality of satellites, or any other dataand/or signals that could be used to determine the current state of thevehicle 100 or determine the position of the vehicle 100 with respect toits environment for use in either creating a map or determining theposition of the vehicle 100 in respect to map data.

The automated driving module(s) 160 either independently or incombination with the detection system 170 can be configured to determinetravel path(s), current autonomous driving maneuvers for the vehicle100, future autonomous driving maneuvers and/or modifications to currentautonomous driving maneuvers based on data acquired by the sensor system120, driving scene models, and/or data from any other suitable sourcesuch as determinations from the sensor data 250. “Driving maneuver”means one or more actions that affect the movement of a vehicle.Examples of driving maneuvers include: accelerating, decelerating,braking, turning, moving in a lateral direction of the vehicle 100,changing travel lanes, merging into a travel lane, and/or reversing,just to name a few possibilities. The automated driving module(s) 160can be configured to implement determined driving maneuvers. Theautomated driving module(s) 160 can cause, directly or indirectly, suchautonomous driving maneuvers to be implemented. As used herein, “cause”or “causing” means to make, command, instruct, and/or enable an event oraction to occur or at least be in a state where such event or action mayoccur, either in a direct or indirect manner. The automated drivingmodule(s) 160 can be configured to execute various vehicle functionsand/or to transmit data to, receive data from, interact with, and/orcontrol the vehicle 100 or one or more systems thereof (e.g., one ormore of vehicle systems 140).

Detailed embodiments are disclosed herein. However, it is to beunderstood that the disclosed embodiments are intended as examples.Therefore, specific structural and functional details disclosed hereinare not to be interpreted as limiting, but merely as a basis for theclaims and as a representative basis for teaching one skilled in the artto variously employ the aspects herein in virtually any appropriatelydetailed structure. Furthermore, the terms and phrases used herein arenot intended to be limiting but rather to provide an understandabledescription of possible implementations. Various embodiments are shownin FIGS. 1-5 , but the embodiments are not limited to the illustratedstructure or application.

The flowcharts and block diagrams in the figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments. In this regard, a block in the flowcharts or block diagramsmay represent a module, segment, or portion of code, which comprises oneor more executable instructions for implementing the specified logicalfunction(s). It should also be noted that, in some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved.

The systems, components, and/or processes described above can berealized in hardware or a combination of hardware and software and canbe realized in a centralized fashion in one processing system or in adistributed fashion where different elements are spread across severalinterconnected processing systems. Any kind of processing system oranother apparatus adapted for carrying out the methods described hereinis suited. A typical combination of hardware and software can be aprocessing system with computer-usable program code that, when beingloaded and executed, controls the processing system such that it carriesout the methods described herein.

The systems, components, and/or processes also can be embedded in acomputer-readable storage, such as a computer program product or otherdata programs storage device, readable by a machine, tangibly embodyinga program of instructions executable by the machine to perform methodsand processes described herein. These elements also can be embedded inan application product which comprises the features enabling theimplementation of the methods described herein and, which when loaded ina processing system, is able to carry out these methods.

Furthermore, arrangements described herein may take the form of acomputer program product embodied in one or more computer-readable mediahaving computer-readable program code embodied, e.g., stored, thereon.Any combination of one or more computer-readable media may be utilized.The computer-readable medium may be a computer-readable signal medium ora computer-readable storage medium. The phrase “computer-readablestorage medium” means a non-transitory storage medium. Acomputer-readable storage medium may be, for example, but not limitedto, an electronic, magnetic, optical, electromagnetic, infrared, orsemiconductor system, apparatus, or device, or any suitable combinationof the foregoing. More specific examples (a non-exhaustive list) of thecomputer-readable storage medium would include the following: a portablecomputer diskette, a hard disk drive (HDD), a solid-state drive (SSD), aROM, an EPROM or Flash memory, a portable compact disc read-only memory(CD-ROM), a digital versatile disc (DVD), an optical storage device, amagnetic storage device, or any suitable combination of the foregoing.In the context of this document, a computer-readable storage medium maybe any tangible medium that can contain, or store a program for use byor in connection with an instruction execution system, apparatus, ordevice.

Generally, modules as used herein include routines, programs, objects,components, data structures, and so on that perform particular tasks orimplement particular data types. In further aspects, a memory generallystores the noted modules. The memory associated with a module may be abuffer or cache embedded within a processor, a RAM, a ROM, a flashmemory, or another suitable electronic storage medium. In still furtheraspects, a module as envisioned by the present disclosure is implementedas an ASIC, a hardware component of a system on a chip (SoC), as aprogrammable logic array (PLA), or as another suitable hardwarecomponent that is embedded with a defined configuration set (e.g.,instructions) for performing the disclosed functions.

Program code embodied on a computer-readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wireline, optical fiber, cable, radio frequency (RF), etc., or anysuitable combination of the foregoing. Computer program code forcarrying out operations for aspects of the present arrangements may bewritten in any combination of one or more programming languages,including an object-oriented programming language such as Java™,Smalltalk™, C++ or the like and conventional procedural programminglanguages, such as the “C” programming language or similar programminglanguages. The program code may execute entirely on the user's computer,partly on the user's computer, as a stand-alone software package, partlyon the user's computer and partly on a remote computer, or entirely onthe remote computer or server. In the latter scenario, the remotecomputer may be connected to the user's computer through any type ofnetwork, including a local area network (LAN) or a wide area network(WAN), or the connection may be made to an external computer (forexample, through the Internet using an Internet Service Provider).

The terms “a” and “an,” as used herein, are defined as one or more thanone. The term “plurality,” as used herein, is defined as two or morethan two. The term “another,” as used herein, is defined as at least asecond or more. The terms “including” and/or “having,” as used herein,are defined as comprising (i.e., open language). The phrase “at leastone of . . . and . . . ” as used herein refers to and encompasses anyand all combinations of one or more of the associated listed items. Asan example, the phrase “at least one of A, B, and C” includes A, B, C,or any combination thereof (e.g., AB, AC, BC or ABC).

Aspects herein can be embodied in other forms without departing from thespirit or essential attributes thereof. Accordingly, reference should bemade to the following claims, rather than to the foregoingspecification, as indicating the scope hereof.

What is claimed is:
 1. A detection system for identifying traffic lightscomprising: a processor; and a memory storing instructions that, whenexecuted by the processor, cause the processor to: estimate, from animage using a first model, depth and orientation information of thetraffic lights relative to a driving lane of a vehicle; compute, using asecond model, relevancy scores for the traffic lights according togeometric inferences between the depth and the orientation information;assign, using the second model, a primary relevancy score for a light ofthe traffic lights associated with the driving lane according to thedepth and the orientation information; and execute a control task by thevehicle according to the primary relevancy score and a state confidence,computed by the first model, for the light.
 2. The detection system ofclaim 1, wherein the instructions to assign the primary relevancy scorefurther include instructions to assign, using the second model, asecondary relevancy score for another light of the traffic lights in anadjacent lane to the vehicle substantially independent of map data. 3.The detection system of claim 1, wherein the state confidence indicatesprobabilities of a color and a shape for the light of the trafficlights.
 4. The detection system of claim 1, wherein the instructions toestimate the depth and the orientation information further includeinstructions to determine, using the first model, location information,wherein the depth, the orientation, and the location informationrepresent geometries of a traffic scene associated with the trafficlights computed from the image and the state confidence representsprobabilities for characteristics associated with the traffic lights. 5.The detection system of claim 4, wherein the geometries of the trafficscene indicate an angle between the traffic lights and the vehiclerelative to the driving lane.
 6. The detection system of claim 1,wherein the primary relevancy score is assigned substantiallyindependent of map data for the traffic lights and the driving lane. 7.The detection system of claim 1, wherein the image is captured by amonocular camera of a single view for the traffic lights.
 8. Anon-transitory computer-readable medium comprising: instructions thatwhen executed by a processor cause the processor to: estimate, from animage using a first model, depth and orientation information of trafficlights relative to a driving lane of a vehicle; compute, using a secondmodel, relevancy scores for the traffic lights according to geometricinferences between the depth and the orientation information; assign,using the second model, a primary relevancy score for a light of thetraffic lights associated with the driving lane according to the depthand the orientation information; and execute a control task by thevehicle according to the primary relevance score and a state confidence,computed by the first model, for the light.
 9. The non-transitorycomputer-readable medium of claim 8, wherein the instructions to assignthe primary relevancy score further include instructions to assign,using the second model, a secondary relevancy score for another light ofthe traffic lights in an adjacent lane to the vehicle substantiallyindependent of map data.
 10. The non-transitory computer-readable mediumof claim 8, wherein the state confidence indicates probabilities of acolor and a shape for the light of the traffic lights.
 11. Thenon-transitory computer-readable medium of claim 8, wherein theinstructions to estimate the depth and the orientation informationfurther include instructions to determine, using the first model,location information, wherein the depth, the orientation, and thelocation information represent geometries of a traffic scene associatedwith the traffic lights computed from the image and the state confidencerepresents probabilities for characteristics associated with the trafficlights.
 12. The non-transitory computer-readable medium of claim 11,wherein the geometries of the traffic scene indicate an angle betweenthe traffic lights and the vehicle relative to the driving lane.
 13. Thenon-transitory computer-readable medium of claim 8, the primaryrelevancy score is assigned substantially independent of map data forthe traffic lights and the driving lane.
 14. A method comprising:estimating, from an image using a first model, depth and orientationinformation of traffic lights relative to a driving lane of a vehicle;computing, using a second model, relevancy scores for the traffic lightsaccording to geometric inferences between the depth and the orientationinformation; assigning, using the second model, a primary relevancyscore for a light of the traffic lights associated with the driving laneaccording to the depth and the orientation information; and executing acontrol task by the vehicle according to the primary relevancy score anda state confidence, computed by the first model, for the light.
 15. Themethod of claim 14, wherein assigning the primary relevancy scorefurther includes assigning, using the second model, a secondaryrelevancy score for another light of the traffic lights in an adjacentlane to the vehicle substantially independent of map data.
 16. Themethod of claim 14, wherein the state confidence indicates probabilitiesof a color and a shape for the light of the traffic lights.
 17. Themethod of claim 14, wherein estimating of the depth and the orientationinformation further includes determining, using the first model,location information, wherein the depth, the orientation, and thelocation information represent geometries of a traffic scene associatedwith the traffic lights computed from the image and the state confidencerepresents probabilities for characteristics associated with the trafficlights.
 18. The method of claim 17, wherein the geometries of thetraffic scene indicate an angle between the traffic lights and thevehicle relative to the driving lane.
 19. The method of claim 14,wherein the primary relevancy score is assigned substantiallyindependent of map data for the traffic lights and the driving lane. 20.The method of claim 14, wherein the image is captured by a monocularcamera of a single view for the traffic lights.